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Study objective: We identify measures of emergency department (ED) and hospital 
workflow that would be of value in understanding, monitoring, and managing crowding. 

Methods: A national group of 74 experts developed 113 potential measures using a 
conceptual model of ED crowding that segmented the measures into input, through- 
put, and output categories. Ten investigators then used group consensus methods to 
revise and consolidate them into a refined set of 30 measures that were rated by all 
74 experts, who used a magnitude estimation technique on a Web site. Each measure 
was compared with a standard to obtain numeric ratings for feasibility, affordability, 
early warning potential, long-term planning potential, a summary rating of operational 
usefulness, and research potential. After review of the comprehensiveness of the re- 
sulting measures, 8 additional measures were developed and also rated by all reviewers. 

Results: The original set of 1 13 measures (46 input, 35 throughput, and 32 output) was 
reduced to 38 through the iterative revision and rating process (15 input, 9 throughput, 
and 14 output). Summary scores in each rating category showed significant variation 
in ratings among the various potential measures. For measures that address similar 
concepts, the priority ranking depended on the rating category chosen. 

Conclusion: The final 38 measures of ED and hospital workflow provide a useful 
pool from which EDs and policymakers can draw to improve their ability to understand 
and address the issue of ED crowding. These measures require rigorous testing for 
feasibility, reliability, and value. 

[Ann Emerg Med. 2003;42:824-834.] 
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Crowding of the nation's emergency departments (EDs) has become a maj or health 
care problem and public issue throughout the past decade. 1 In part, this problem has 
resulted from greatly increased patient demand, but it has also been associated with 
decreased hospital capacity, and both of these factors have complex causes. 2,3 
Regardless of the causes, the effects are increasingly troublesome, including diversion 
of ambulances, long patient waits, frustration for patients and ED personnel, greater 
risks for poor outcomes, and unnecessarily high costs. 1,2,4 A recent survey in 
California found that all ED directors at university or county hospitals and 96% of 
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Capsule Summary 

What is already known on this topic 

Although emergency department (ED) crowding is a topic of 
increasing public and professional concern, there is no standard- 
ized definition of it and little agreement on what factors may 
contribute to it. 

What question this study addressed 

To use a broad-based and thorough expert process to identify all 
measures of ED and hospital workflow that may be useful in 
understanding, monitoring, and managing crowding. 

What this study adds to our knowledge 

A panel of 74 national experts assessed 113 measures, and chose 
38 through a discussion and rating process. 

How this might change clinical practice 

The 38 measures should serve as a resource for research to 
determine which ones are related to crowding, and eventually to 
develop tools to predict and modify crowding. 



those at private or community hospitals reported crowd- 
ing as a problem, and 28% said it was a daily occurrence. 5 

Many factors contribute to ED crowding, but there is 
relatively little empiric research on this topic. The 
General Accounting Office recently reported the results 
of a 2-year nationwide study of the problem. 6 They 
found that the inability to transfer admitted ED patients 
to inpatient beds was the factor most commonly associ- 
ated with crowding. Although the contributing reasons 
for this lack of inpatient beds are complex, the General 
Accounting Office identified 2 primary causes: (1) hos- 
pitals have strong economic incentives to staff only 
beds that will nearly always be full (which impairs their 
ability to respond to intermittent surges in demand) ; 
and (2) ED patients must compete with other sources of 
inpatient admissions, many of which generate higher 
revenue for hospitals (eg, elective surgical procedures, 
cardiac catheterization) . Other suggested causes of 
crowding include an increase in ED visits and in sever- 
ity, increased extensive therapy in the ED, difficulty 
obtaining timely consultations, inefficient surgical 
scheduling, the nursing shortage, the uninsured, delayed 
access to ancillary services, reduced availability of sub- 
acute and long-term-care beds, increased operational 
costs, natural fluctuations in demand, reduced hospital 
and on-call specialist reimbursements, hospital re- 
structuring, and changing patient demographics. 1 ' 5 ' 7 " 10 

Schull et al 1 1 assembled an expert panel of 10 front- 
line key informants from 4 hospitals and an ambulance 
service in Canada to develop a standard definition of ED 
crowding and a list of key determinants thought to be 



most important. They decided on ambulance diversion 
as an appropriate operational definition and proxy mea- 
sure of urban ED crowding and identified 25 factors as 
potentially important determinants of crowding. What- 
ever the causes, it seems likely that a comprehensive set 
of measures would help in understanding which factors 
contribute the most to this situation and to assist with 
monitoring and managing it. Focusing on some high- 
priority measures might also stimulate others to create 
obj ective data about this problem, which has had much 
press and public attention but little quantitative infor- 
mation. 

Before comprehensive measurement systems for ED 
crowding could be developed, it was necessary to have a 
conceptual model for the various factors that may con- 
tribute to the problem. Recently, Asplin et al 12 have pro- 
posed a conceptual model for ED crowding (Figure) that 
is described in more detail elsewhere. The model is 
based on engineering principles from queuing theory 
and compartmental models of flow, dividing ED func- 
tion into input, throughput, and output stages. 13 These 
concepts have been used extensively in health care to 
understand and improve hospital bed allocation, oper- 
ating room staffing, coronary angiography services, pri- 
mary care access for appointments, and a variety of 
medical systems, as well as in clinical toxicology, 
pharmacokinetics, and the like. 1417 Similar concepts 
were also suggested specifically for ED patient flow by 
Coats and Michalis 18 and by Litvak et al. 9 

The input-throughput-output model permits most 
factors affecting use and crowding to be grouped into 1 
of these 3 stages. 12 Thus, input or demand for ED ser- 
vices depends on the volume of ill and injured people in 
the community and the capability of the rest of the 
health care system to address the needs of individuals 
not requiring emergency care. Throughput includes 
factors that affect the efficiency of an ED to cope with its 
input, ranging from ED beds and staffing to the effi- 
ciency of ancillary services and consultant access. 
Finally, output factors include the ability of the inpa- 
tient system to admit patients requiring hospital care 
and of the ambulatory care system to provide timely 
postdischarge care (Figure). 

With such a model, addressing the measurement 
issue comprehensively becomes possible. The goal of 
this project was to develop measures that could be used 
for monitoring, planning, and research. This article 
describes the development of these measures, using the 
input-throughput-output conceptual model as a way of 
grouping and applying them for various uses. 
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MATERIALS AND METHODS 

We assembled a core group of investigators to develop, 
refine, and modify potential measures, and a much 
larger group of experts in various related fields who 
were willing to serve as reviewers for any proposed 
measures. We (BRA, DJM) directly recruited the core 
investigators and reviewers in response to a request for 
task order proposals from the Agency for Healthcare 
Research and Quality The task order request instructed 
responding organizations to "select a group of content 
experts with expertise representing clinical care, data, 
emergency medical services, ED staff, hospital adminis- 
tration, information technology, and other relevant 
areas." We contacted potential reviewers from academic 
departments of emergency medicine, the American 
College of Emergency Physicians, the American Hospital 
Association's Task Force on Coverage and Access (200 1 ) , 
the HMO Research Network, expert panel meetings on 
ED crowding sponsored by the Agency for Healthcare 
Research and Quality and the Robert Wood Johnson 
Foundation, and from our own organizations. After the 



task order proposal was funded, we recruited additional 
reviewers through an open invitation to attendees at the 
2002 annual meeting of the Society for Academic Emer- 
gency Medicine. Our goal was to recruit a group of 
expert reviewers with varied expertise and experience 
who were independent of the core investigator group. 
Everyone who was asked to be part of the core group 
agreed, and fewer than 10 who were asked to be review- 
ers declined, usually because of being too busy. The 
final group of reviewers includes experts from 58 orga- 
nizations in 21 states; the majority (72%) are emer- 
gency physicians (see the acknowledgments for identi- 
fication of these people and their positions). 

The 10 core investigators (the authors and 6 others 
noted in the acknowledgments) and the 64 reviewers 
have collective expertise in ED and hospital administra- 
tion, clinical care (including nursing), research, emer- 
gency medical services (EMS) , health system leader- 
ship, and information technology. We also attempted to 
achieve diversity among the reviewers in hospital type 
(eg, teaching versus nonteaching) and geographic 
region. The procedures followed were approved by the 



Figure. 



Conceptual framework for ED workflow and crowding. Reprinted from Asplin BR, Magid Df, Rhodes KV, et al. A conceptual model 
of emergency department crowding. Ann Emerg Med. 2003;42:173-180 with permission from American College of Emergency 
Physicians. 12 



Input 



Emergency care 

• Seriously ill and injured 
patients from the community 

• Referral of patients with 
emergency conditions from 
other providers 



Unscheduled urgent care 

• Lack of capacity for 
unscheduled care in the 
ambulatory care system 

• Desire for immediate care 
(eg, convenience, conflicts 
with job, family duties) 



Safety net care 
•Vulnerable populations 
(eg, Medicaid beneficiaries, 
the uninsured) 
•Access barriers (eg, 
financial, transportation, 
insurance, lack of usual 
source of care) 



Throughput 



Output 



Ambulance 
diversion 



Demand for 
ED care 



Patient arrives at ED 



Triage and room 
placement 



Diagnostic evaluation 
and ED treatment 



ED boarding of inpatients 



Lack of access to follow-up care 



Leaves without 
treatment 
complete 



Patient 
disposition 



Lack of available 
staffed inpatient beds 



ACUTE CARE SYSTEM 



Ambulatory 
care 
system 



a: 



Transfer to other 
facility (eg, skilled 
nursing, referral 
hospital) 



Admit to 
hospital 
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HealthPartners institutional review board. A modifica- 
tion of the anonymous rating of the Delphi method and 
the structured meeting process of nominal group tech- 
nique was used to obtain consensus among this large 
and diverse group of experts. 19 

The authors first worked with the other core investi- 
gators to establish the rating categories for measures. 
These categories and their final definitions are as follows: 

1 . Feasibility: How feasible would it be for opera- 
tional staff to collect the data needed for this measure 
routinely (or as frequently as would be needed) in the 
rater's ED system or in one known to the rater? 

2. Early warning potential: How well would this 
measure provide warning about impending capacity 
problems within the next 2 to 24 hours? 

3. Planning value: How well would this measure pro- 
vide information about trends and changes in ED business 
and crowding throughout a period of weeks to months? 

4. Cost-efficiency: How affordable would the data 
collection be for this measure? 

5 . Summary rating of operational usefulness: Ac- 
cording to a combination of the above criteria, how use- 
ful would this measure be for clinical and administra- 
tive operations? 

6. Usefulness for research: Entirely apart from the 
aforementioned criteria, how much would this measure 
help to improve our general understanding of the causes 
and consequences of ED crowding? 

The next step was to ask the core investigators and 
the reviewers to submit individual measures for each 
of the stages of the input- throughput-output concep- 
tual model, including measures that were derived 
from the literature, in use in their own organizations, 
or newly created to address a potential causative fac- 
tor. Any measure proposed required an operational 
definition, rationale, and the data elements needed to 
calculate it. One hundred thirteen individual mea- 
sures were proposed by various members of this group 
(46 input, 35 throughput, and 32 output) . Exact sub- 
mission sources are unknown because submitted mea- 
sures were stripped of identifiers to maintain as much 
confidentiality and objectivity as possible. These 113 
measures were reduced by the authors to 77 (3 1 input, 
22 throughput, and 24 output) by eliminating dupli- 
cate or overlapping measures and those that did not 
meet the criteria adequately. The core investigator 
group then reviewed these proposed measures during 
a 2-day meeting. This meeting was conducted by a 
facilitator (BRA) who actively sought (and obtained) 
input from all of the attendees about any changes in 



the wording, feasibility, potential value for under- 
standing ED crowding, and overlap with other mea- 
sures. During this process, every measure was 
changed, some were combined, and those of limited 
value were eliminated, reducing the measures to 30 
(12 input, 7 throughput, and 11 output). 

This revised set of measures was then rated by 56 of 
the 64 reviewers and the core investigators on an Inter- 
net Web site by using a magnitude estimation tech- 
nique. 20 ' 21 This technique permits averaging of ratings 
across many raters on a ratio level scale by asking each 
respondent to provide a relative score from 0 to infinity 
for each item in comparison with a measure used as a 
standard. The standard score was set at 100. For exam- 
ple, if the feasibility of a measure was believed to be 
twice that of the standard in the mind of a reviewer, a 
score of 200 would be assigned. Likewise, if it were half 
as feasible, the reviewer would assign a score of 50. 
Theoretically and empirically, the distribution of scores 
from a magnitude likelihood task are log linear, and 
thus the geometric mean rather than the more common 
arithmetic mean is the appropriate measure of central 
tendency, which results in much less clustering of 
scores than often occurs with rating scales using the 
more traditional Likert scale. It also makes it easier to 
interpret the ratings because a rating of 200 for a mea- 
sure means that the reviewers as a group thought that 
the measure was literally twice as good as one that was 
rated 100 in the same category. 

The ratings and associated comments were summa- 
rized and reviewed during a second 2-day meeting of 
the core investigator group. These discussions led the 
group to identify 8 additional measures to adequately 
cover all of the concepts that they believed were impor- 
tant. These additional measures (3 input, 2 throughput, 
and 3 output) were then also subjected to the same rat- 
ing system on the Web site by 50 reviewers and the core 
investigators. 



RESULTS 

The final 38 measures and their operational definitions 
are described in Tables 1,2, and 3 , separated by the 3 
stages of our conceptual framework. An Appendix con- 
taining each measure's operational definition, ratio- 
nale, and required data elements is available online at 
Annals' Web site (www.mosby.com/AnnEmergMed). To 
clarify their purpose, the measures have been grouped 
within each stage by the main concept they represent. 
These concepts and their definitions are as follows: 
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Table 1. 






Input measures. 






Input Measure 


Concept 


Operational Definition 



1 . ED patient volume, standardized for bed hours Patient demand 

2. ED patient volume, standardized for annual average Patient demand 

3. ED ambulance patient volume, standardized for bed hours Patient demand 

4. ED ambulance patient volume, standardized for annual average Patient demand 

5. Patient source Patient demand 

6. Percentage of open appointments Patient demand 

7. Percentage of patients who leave without treatment complete* ED capacity 

8. Leave without treatment complete severity* ED capacity 

9. Ambulance diversion episodes ED capacity 

10. Ambulance diversion requests denied and forced openings ED capacity 

11. Diverted ambulance patient description ED capacity 

12. Average EMS waiting time ED efficiency 

13. Patient complexity as assessed at triage Patient complexity 

14. Patient complexity as the percentage of ambulance patients Patient complexity 

15. Patient complexity as assessed by coding Patient complexity 



Number of new patients registered within a defined period (hour, shift, 

day) number of ED bed hours within this period 
Number of new patients registered within a defined period -f annual 

mean number new patients registered within this period 
Number of new ambulance patients registered within a defined period -f 

number of ED bed hours within this period 
Number of new ambulance patients within a defined period -f annual 

average of new ambulance patients registered within this period 
Time, arrival mode, reason, referral source, and usual care for each 

patient registering at an ED in a defined period (hour/shift/day) 
Percentage of open appointments at the beginning of a day in 

ambulatory care clinics that serve an ED's patient population 
Number of registered patients who leave the ED without treatment 

complete -f total number of patients who register during this period 
Average severity of patients who leave the ED without treatment 

complete within a defined period (shift/day/week) 
Number and duration of all diversion episodes at EDs within the EMS 

system within a defined period (week/month/year) 
Number of diversion requests denied or forced openings within a 

defined period (week/month/year) 
Chief complaints and final destination of diverted EMS patients within a 

defined period (week/month/year) 
Total time at hospital for ambulances delivering patients to ED during a 

defined period (shift/day/week/month) -f number of ambulance 

deliveries within that period 
Mean complexity level as assessed at triage (using local criteria) for all 

patients triaged in a defined period (shift/day/week/month) 
Percentage of patients registering at an ED in a defined period 

(shift/day/week/month) who arrived by ambulance 
Mean complexity level as coded at the end of the visit for all patients 

completed in a defined period (shift/day/week/month) 



*Leave without treatment complete includes those patients who leave without being seen, leave before being finished, and leave against medical advice. 



Concept Operational Definition 



Table 2. 

Throughput measures. 



Throughput Measure 



1. ED throughput time 


ED efficiency 


2. ED bed placement time 


ED efficiency 


3. ED ancillary service turnaround time 


ED efficiency 


4. Summary workload, standardized for ED bed hours 


ED workload 


5. Summary workload, standardized for registered nurse staff hours 


ED workload 


6. Summary workload, standardized for physician staff hours 


ED workload 


7. ED occupancy rate 


ED workload 


8. ED occupancy 


ED workload 


9. Patient disposition to physician staffing ratio 


ED workload 



Average time between patient sign-in and departure (separately for ad- 
mitted vs discharged patients) within a defined period (day/week/month) 

Mean interval between patient sign-in and placement in a treatment 
area within a defined period (shift/day/week/month) 

Average time between physician order and result report (separately for 
each service area) within a defined period (shift/day/week/month) 

Summary of (patients treated x acuity) in a defined period (shift/day/ 
week) number of ED bed hours within this period 

Summary of (patients treated x acuity) in a defined period 
(shift/day/week) total ED staff registered nurse hours within this period 

Summary of (patients treated x acuity) in a defined period (shift/day/ 
week) total ED staff physician hours within this period 

Total number of ED patients registered at a defined time -f number of 
staffed treatment areas at that time 

Total number of patients present in the ED at a defined time -f number of 
staffed treatment areas at that time 

Number of patients admitted or discharged per staff physician during a 
defined period (shift/day/week) 
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1 . Patient demand (6 items) : The volume of patients 
presenting to the ED for medical care. 

2. Patient complexity (3 items): Patient factors such 
as the urgency and potential seriousness of the present- 
ing complaint, the stability of the clinical condition, 
and the baseline medical and psychosocial burden of ill- 
ness. 

3 . ED capacity (6 items) : The ability of the ED to pro- 
vide timely care for the level of patient demand accord- 
ing to the adequacy of physical space, equipment, per- 
sonnel, and the organizational system. 

4. ED workload (6 items) : The demand and complex- 
ity of patient care that is undertaken by the ED within a 
given period. 

5 . ED efficiency (3 items) : The ability of the ED to 
provide timely, high-quality emergency care while lim- 
iting waste of equipment, supplies, and effort. 

6. Hospital capacity (6 items) : The ability of the hos- 
pital to provide timely inpatient care for ED patients 
who require hospitalization according to the adequacy 



of physical space, equipment, personnel, and the orga- 
nizational system. 

7. Hospital efficiency (8 items) : The ability of the 
hospital to provide timely, high-quality inpatient care 
while limiting waste of equipment, supplies, and effort. 

The core investigators thought that it was important 
to provide reviewers and potential users with the option 
of several approaches to measures that required stan- 
dardization. Therefore, some of the measures are simi- 
lar (eg, ED patient or ambulance volume standardized 
in bed hours versus annual average in input [Table 1 ] 
and summary workload measure standardized in 3 ways 
in throughput [Table 2]). 

Tables 4 to 6 display the relative scores and rankings 
for each measure in relation to summary usefulness and 
to each of the 5 subcategories. In each table, the mea- 
sure scoring 1 00 points was the one that was chosen as 
the standard for comparative rating of all the others in 
that portion of the framework. The measures have been 
ranked in order by their overall operational summary 



Table 3. 

Output measures. 



Output Measure Concept Operational Definition 



1 . ED boarding time Hospital efficiency Mean time from inpatient bed request to physical departure of patients from the ED overall 

and by bed type within a defined period (shift/day/weekf 

2. ED boarding time components Hospital efficiency Mean time from inpatient bed request to physical departure of patients from the ED by bed 

type by component (bed assignment, bed cleaning, transfer arrival) within a defined period* 

3. Boarding burden Hospital efficiency Mean number of ED patients waiting for an inpatient bed within a defined period 4- number 

of staffed ED treatment areas 

4. Hospital admission source, standardized Hospital efficiency Number of requests for admission within a defined period (shift/day) overall and by 

and adjusted admission source 4- annual mean requests for admission during that period and source 

and adjusted for day of week and season of year 1 

5. ED admission transfer rate Hospital efficiency Number of patients transferred from ED to another facility who would normally have been 

admitted within a defined period -f number of ED admissions within this period 

6. Hospital discharge potential Hospital efficiency Number of inpatients ready for discharge at or within a defined period -f number of 

hospital inpatients at that time 

7. Hospital discharge process interval Hospital efficiency Mean interval from discharge order to patient departure from a unit in a defined period 

(shift/day/week/month) 

8. Inpatient cycling time Hospital efficiency Mean amount of time required to discharge an inpatient and admit a new patient to the 

same bed within this period 

9. Hospital census Hospital capacity Mean number of inpatient beds available by bed type at a defined time -f number of staffed 

inpatient beds by bed type* 

10. Hospital occupancy rate Hospital capacity Number of occupied inpatient beds overall and by bed type -f number of staffed inpatient 

beds overall and by bed type* 

1 1 . Hospital supply/demand status forecast Hospital capacity Forecast of expected hospital admissions and discharges as reported daily at 6 am and 

compared with hospital census 

12. Observation unit census Hospital capacity Mean number of available ED observation beds at a defined time -f number of staffed ED 

observation beds 

13. ED volume/hospital capacity ratio Hospital capacity Number of new ED patients within a defined period (shift/day) number of available 

hospital beds at the beginning of analysis period overall and by bed type* 

14. Agency nursing expenditures Hospital capacity Registered nurse agency nursing expenditures (ED/overall) within a defined period 4- total 

nursing expenditures (ED/overall) within this period 

*Bed type=ICU/telemetry/psychiatry/ward. 

Admission source=ED/operating room/catheterization laboratory/outpatient/other. 
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score. Although about half of the measures have only 
small variation in their rankings relative to other mea- 
sures across all 6 categories, the rest display larger vari- 
ations. The categories causing the greatest degree of 
variance among measures were feasibility and research 
potential. 

Among the input measures in Table 4, there was a 
strong preference for choosing patient volume stan- 
dardized by bed hours over that by annual average, and 
it was the favored way to measure the concept of patient 
demand. Although there was much less overall prefer- 



ence to have a patient complexity measure assessed by 
coding data than by triage or the percentage of ambu- 
lance patients, for research purposes, the 3 measures 
have equivalent perceived value. The most striking 
variance within a measure is for patient source. Although 
it was ranked relatively low for all other categories, it 
received the highest score of any input measure for re- 
search purposes. 

None of the throughput measures in Table 5 ranked 
high for feasibility or cost-efficiency, but most of them 
nevertheless received high scores for overall value, plan- 



Table 4. 

Ratings for various aspects of input measures in order of summary value. 



Mean, Rank 



Early Cost- 
Input Measure Concept Summary Warning Planning Feasibility efficiency Research 



ED patient volume, standardized for bed hours 


Patient demand 


201,1 


168,1 


196,1 


170,1 


148,4 


190,2 


Ambulance diversion episodes 


ED capacity 


176,2 


82,7 


181,2 


145,6 


141,7 


190,2 


Percentage of patients who left without treatment complete* 


ED capacity 


163,3 


66,10 


161,4 


157,4 


155,3 


180,4 


ED ambulance patient volume, standardized for bed hours 


Patient demand 


161,4 


138,3 


163,3 


153,5 


142,6 


162,6 


ED ambulance patient volume, standardized for annual average 


Patient demand 


148,5 


119,5 


142,7 


160,3 


158,2 


150,8 


Patient complexity as assessed at triage 


Patient complexity 


148,5 


130,4 


147,6 


139,7 


143,5 


164,5 


Patient complexity as the percentage of ambulance patients 


Patient complexity 


148,5 


147,2 


148,5 


165,2 


160,1 


145,9 


Ambulance diversion requests, denied and forced openings 


ED capacity 


108,8 


71,9 


118,9 


79,10 


105,9 


123,10 


ED patient volume, standardized for annual average 


Patient demand 


100,9 


100,6 


100,12 


100,9 


100,10 


100,12 


Average EMS waiting time 


ED efficiency 


95,10 


72,8 


112,11 


72, 11 


68,11 


97,13 


Patient source 


Patient demand 


87,11 


33,12 


116,10 


52,13 


44,14 


198,1 


Patient complexity as assessed by coding data 


Patient complexity 


76,12 


3,15 


137,8 


138,8 


126,8 


154,7 


Diverted ambulance patients 


ED capacity 


74,13 


25,14 


77,13 


44,14 


53,13 


110,11 


Left without treatment complete severity* 


ED capacity 


64, 14 


39,11 


72, 15 


66,12 


66,12 


83,14 


Percentage of open appointments 


Patient demand 


39,15 


26,13 


74,14 


12,15 


18,15 


62,15 



*Left without treatment complete includes patients who leave without being treated, without being finished, and against medical advice. 



Table 5. 

Ratings for various aspects of throughput measures in order of summary value. 



Mean, Rank 



Early Cost- 
Throughput Measure Concept Summary Warning Planning Feasibility efficiency Research 



ED throughput time 


ED efficiency 


195,1 


110,6 


206,1 


102,4 


100,5 


212,1 


ED occupancy 


ED workload 


182,2 


184,1 


182,4 


144,1 


109,3 


170,5 


Summary workload, standardized for ED bed hours 


ED workload 


170,3 


139,3 


188,2 


73,9 


76,9 


195,2 


ED bed placement time 


ED efficiency 


162,4 


159,2 


177,5 


123,2 


116,1 


162,6 


Summary workload, standardized for registered nurse staff hours 


ED workload 


161,5 


127,5 


185,3 


84,8 


85,7 


175,3 


Summary workload, standardized for physician staff hours 


ED workload 


156,6 


129,4 


165,6 


93,6 


103,4 


175,3 


Patient disposition to physician staffing ratio 


ED workload 


138,7 


102,7 


139,8 


117,3 


112,2 


146,7 


ED ancillary service turnaround time 


ED efficiency 


132,8 


63,9 


140,7 


88,7 


82,8 


129,8 


ED occupancy rate 


ED workload 


100,9 


100,8 


100,9 


100,5 


100,6 


100,9 
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ning potential, and research utility. ED throughput time 
was clearly the most favored measure of ED efficiency It 
was believed that the ED workload was best measured by 
the ED occupancy measure and the summary workload 
measure that was standardized by ED bed hours. 

Among the output measures in Table 6, ED boarding 
time and boarding burden were the most favored ways to 
measure hospital efficiency and mean hospital discharge 
process interval was the preferred measure of the dis- 
charge process. Three of the 6 measures of hospital capac- 
ity were favored and attained relatively similar scores. 



LIMITATIONS 

Several issues or limitations should be taken into ac- 
count by anyone who is considering using these mea- 
sures. Most obvious is the need for rigorous testing for 
actual feasibility cost, value, and reliability before 
widespread use. Although some measures were rated 
highly on feasibility and cost-efficiency by our review- 
ers, formal testing of the measures is still warranted. 
Those who conduct such tests need to consider care- 
fully how obtainable these measures are and in what 
proportion and types of EDs they could be collected. 
Once such testing has been done, we also need to learn 
just how feasible the findings are for early warning, 
long-term planning, or research purposes. 

Some readers may be concerned that leaving some 
measures to variable local criteria (eg, patient complex- 
ity) or leaving out risk adjustment makes it difficult to 



use the measures to compare different EDs. Such stan- 
dardization would indeed be necessary for that pur- 
pose; however, we did not create the measures for that 
reason. We wanted measures that would provide infor- 
mation for a wide range of purposes. In particular, we 
sought measures that would provide early warning about 
impending crowding, information for long-range plan- 
ning about patient flow and capacity needs, and new 
data for the crowding research agenda. As the feasibility 
and usefulness of these measures is tested during imple- 
mentation, the performance of these and other stan- 
dardization and risk adjustment methodologies should 
be formally evaluated and reported. 

A potential problem for the usefulness of these mea- 
sures is that they mostly track activities over time, 
whereas ED workflow is often disrupted by brief peri- 
ods of heavy or light load on its input, throughput, or 
output. Also, a few cases of extraordinary difficulty in 
medical or psychosocial care can have a dispropor- 
tionate effect on ED workflow. The proposed measures 
may not completely capture the effect of such chal- 
lenging medical cases on ED crowding. Finally, geo- 
graphic variation in the causes and consequences of 
ED crowding may be significant, and the proposed mea- 
sures may not capture this variation comprehensively. 



DISCUSSION 

The obj ective of this proj ect was to develop measures 
that could be used for early warning, planning, or 



Table 6. 

Ratings for various aspects of output measures in order of summary value. 



Mean, Rank 



Early Cost- 
Output Measure Concept Summary Warning Planning Feasibility efficiency Research 



ED boarding time 


Hospital efficiency 


256,1 


229,1 


256,1 


128,4 


126,4 


275,1 


Boarding burden 


Hospital efficiency 


239,2 


216,2 


237,2 


144, 1 


129,2 


258,2 


Hospital occupancy rate 


Hospital capacity 


203,3 


213,3 


214,3 


131,3 


143,1 


203,3 


Hospital supply/demand status forecast 


Hospital capacity 


191,4 


205,4 


193,4 


116,6 


121,5 


167,6 


ED volume/hospital capacity ratio 


Hospital capacity 


185,5 


167,5 


192,5 


110,9 


105,10 


197,4 


Hospital admission source, standardized and adjusted 


Hospital efficiency 


177,6 


135,6 


190,6 


115,7 


107,8 


196,5 


Mean hospital discharge process interval 


Hospital efficiency 


143,7 


102,7 


164,7 


103,10 


106,9 


151,8 


Observation unit census 


Hospital capacity 


127,8 


85,10 


122, 11 


136,2 


121,5 


125, 11 


ED boarding time components 


Hospital efficiency 


113,9 


90,9 


129,8 


33,14 


37,14 


127,10 


ED admission transfer rate 


Hospital efficiency 


113,9 


35,12 


129,8 


114,8 


108,7 


130,9 


Hospital discharge potential 


Hospital efficiency 


108,11 


63,11 


125,10 


52,13 


128,3 


164,7 


Inpatient cycling time 


Hospital efficiency 


104, 12 


21,13 


120,12 


53,12 


53,12 


109,12 


Hospital census 


Hospital capacity 


100,13 


100,8 


100,13 


100,11 


100,11 


100,13 


Agency nursing expenditures 


Hospital capacity 


82, 14 


13, 14 


83,14 


124,5 


53,12 


57, 14 
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research about the increasingly important health care 
issue of ED crowding. The 38 measures in these tables 
were developed and prioritized by a large and varied 
group of experts through an iterative consensus and rat- 
ing process. Some of these measures have been used by 
individual EDs and by some research studies, but this is 
the first attempt to develop a comprehensive set of mea- 
sures guided by a conceptual framework for operational 
and research purposes. Although many of the measures 
are new, the rigorous consensus process used provides 
some assurance that they warrant consideration and 
further study by others interested in ED function. Health 
care systems differ in the type of data they routinely col- 
lect as part of their daily operations. The applicability 
and generalizability of our results is enhanced by our 
efforts to identify multiple potential measures for each 
ED crowding-related concept. 

Although we have not been able to find any compara- 
ble conceptual models for ED crowding, the panel of 
Schull et al 1 1 did organize factors postulated to be asso- 
ciated with ED ambulance diversion into 4 groups: 
community, patient, ED, and hospital. That study and 
the focus group study by Estey et al 22 also represent 
efforts to assemble front-line emergency care people to 
develop and organize thinking about ED crowding. 

What should the reader make of the rating scores and 
relative rankings for the measures? Both simply provide 
a rough comparison among the various measures within 
each framework stage, concept, and purpose. Because 
the 3 stages were rated with separate reference standard 
measures, it is not possible to directly compare the 
scores across stages. However, the ratings can be com- 
pared within a given stage of the model. It is also not 
appropriate to conclude that a measure with a low score 
in a particular stage is therefore unimportant. Because 
measures address different concepts and stages of ED 
crowding, a relatively low-scoring measure might still 
be needed because its concept is important for a thor- 
ough understanding or for a particular purpose. 

There have been few previously published attempts to 
devise measures for ED crowding. Reeder and Garrison 23 
recently proposed 4 measures: 

1 . Bed ratio= (current ED patients + predicted 
arrivals - predicted departures) treatment areas 

2. Acuity ratio = average triage category of all patients 
in the ED 

3. Provider ratio = arrivals/h -=- sum of average 
patients/h disposed by existing providers 

4. Demand value = bed ratio x provider ratio acuity 
ratio 



Although elements of these measures are included in 
the current set, Reeder and Garrison 23 depend on pre- 
dictions and averages and do not identify the proximate 
causes of crowding. Moreover, this set apparently has 
not been widely applied in practice. 

Another measurement approach was created by 
Weiss et al 24 and consisted of independent observer 
counts of the numbers of patients at various times in an 
academic center ED while the charge nurse and emer- 
gency physicians were simultaneously surveyed. They 
found significant correlations between nurse and 
physician feelings of crowding and being rushed and 
the numbers of patients in various stages of being treated. 
However, this approach does not provide enough detail 
to clarify the reasons for the crowding or to develop 
interventions or policy changes to reduce crowding. 

The measures developed in this study show great 
promise for assisting in the management or prevention 
of ED crowding. A subgroup of these measures should 
be developed for early warning or long-term planning 
and then tested to determine its value for these pur- 
poses. These measures or their associated monitoring 
sets also should permit us to conduct important re- 
search studies to clarify the relationship between ED 
crowding and quality of care. It is widely assumed that 
ED crowding leads to serious patient safety and out- 
comes problems, but thus far we have lacked the tools 
to conduct such studies. 1,2,4 

In the absence of a good set of measures and the re- 
sulting data that could provide a more complete assess- 
ment of the relative importance of various factors that 
affect crowding, we are at risk of seeing poorly designed 
or misguided interventions. Some of these interven- 
tions could conflict with the professionalism and ser- 
vice ethic of emergency providers, antagonize patients, 
or even make matters worse. The measures that are 
defined and rated here require testing and use for opera- 
tional and research purposes, but they represent the 
beginning of a process that can provide important infor- 
mation to those who are in positions to help solve the 
problem of ED crowding. 
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